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ABSTRACT . ; 

..This report describes the initial objectives hnd 
several^modifications*in a project to explore and map the'research 
information, products, and strategies which have implications for 
educational policy. Throe additional end products N were projected: (1) 
Unscreened Priority Area Policy Interpretation Bibliographies, (2) 
Annotated Bibliographies of Priority Areas, and (3) Final Analysis of 
Strengths, Weaknesses, and Gaps. The changes in the project emRhasis 
llilt -the- evaluation of^the effectiveness of the search and analysis 
methods initially employed-. However, general evaluative comments are 
provided, (STS) „ . , - * * * 
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' CEPM PINAL REPORT 


•.Introduction 


r ' • . 


- This brief paper summarizes the tasks and methodology used in carrying 
put 'th® activities and completing the deliverables detailed .in the subcontract 
-proposal and later in the proposed modification. These are discussed in 
chronological order. - 

• The CEPM'subcontract was developed to meet objectives 1 and 2 of the 
R^ifearch and Policy Interpretation proposal: 

Objective 1: To build a collection of exemplary research interpretation 
products and strategies with implications for educational policy. 

Objective 2: To contribute to development of Dissemination/Feedforward 
System through analysis of strengths arid weaknesses and gaps in avail¬ 
able products and strategies and) the implications of this analysis for 
the System. • 

DELIVERABLES ’ * 

• • 

Deliverable 1: Initial Search Strategy . An initial search strategy to 
identify a broad range of synthesis’and interpretation documents was 
distributed to contractors at the first System meeting. No time was 
’ available for discussion and suggestions from other contractors, and 

the strategy was implemented as presented. 

* 

.Deliverable 2: Preliminary Analysis . Abstracts of the documents wenp 
screened for appropriateness and p. list of materials tfi be collected 
. Was generated. References from some of these documents and frqm in- 
house CEPM Consultants led to other print and non-print materials^. At 
the same time,, a survey of some relevant literature and work with in- 
house CEPM consultants led to development of a preliminary conceptual 
framework based upon types of syntheses available and the predicted 
usefulness of these materials to educational policy makers and decision 
makers. Thd list of materials available and the preliminary conceptual 
framework were used to make the preliminary analysis of strengths, 

V „ weaknesses, and gaps. This analysis-presented a broad map of th» 

available materials regarding educational research- syntheses and analyses 
and policy interpretations. The-preliminary analysis was distributed 
. to system contractors. ' ' ' I 


NOTE ON MODIFICATION OF SCOPE OF WORK ' • 

As CEPM work proceeded to ident/fy the policy areas to be' studied m greater 
depth, the focus of national ccfitractors was reconceptualized. This resulted 
in a decreased emphasis on policy interpretations. In addition. Regional 
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*• _ * 

Exchanges identified priority areas inconsistent with CEPM’s identification. 

Thus, CfePM requested a modification of the remaining scope of work in PHa^e I 
to, allow fior an intensive search in the RDX-identified priority areas. In 
order to "retain the work already completed-, CEPM proposed an additional 
deliverable, the unscreened policy-area bibliographies. 


Deliverable 3 




: Unscreened Priority Area Policy Interpretation 
Bibliographies. On.the basis of modified Delphi-consultation with CEPM 
instructional and research faculty, and with state education agency 
staff, a number of priority topic areas were identified. An intensive 
sdarch of materials identified'in^the initial search and of additional 
references produced a list of materials to be purchased. Many of the 
print materials were available in the CEPM library 1 or the University of, 
Oregon library. ' Other materials were purchased. These materials are . 
listed in the,Search Category Bibliographies. Available time and funds 
precluded any analysis of these materials. ■ 


Deliverable 4: Annotated Bibliographies of Priority Areas. An ERIC 
search, a Library of Congress search, and consultation with CEPM and 
University faculty generated a list of materials for consideration. 

These were screened for usefulness to educational practitioners and 
decision makers and a standard format was completed for each entry^ ^ 
included in the three annotated bibliographies. 

Deliverable 5: Final Analyses of Strengths, Weaknesses and Gaps . 

The entries in the Annotated bibliography for each of the three priority 
areas were used in the analyses of strengths, weaknesses and gaps, and 
the indicati&n of implication^ for-the .system.-* 

v * 

NOTE: The annotated bibliography and final analysis for each of the three 
priority areas may be used separately although the two papers for all three 

areas are compiled as a package for submission to the RDX contractors., 

♦ » % 

EVALUATION OF METHODOLOGIES . 

■* - . | I 

The change in intensive search topic areas midway through the supple¬ 
mental search limits our ability to.judge the effectiveness of the method of 
identifying research,and policy interpretations materials. As this change 
led to discontinuingwork on a conceptual framework, our evaluation of*this 
•means of analysis is also limited. However, the following evaluative comments 
are included for any help they might provide to others in designing a search 
strategy. 

Search Methodology 

1. On-line computer search of ERIC (RIB and-CIJE) . Although we.djg^xgned 
the initial search strategy including selection of the descriptors in con¬ 
sultation with a librarian training in on-line computer searching, we obtained 
a fairly high percentage of inappropriate "hits". This fact*, together with 
' later reference, to appropriate materials not identified, leads us to question 
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^the validity t>f the on-line computer search unless it is supplemented by 
otHkr seatch strategies. We have-no feeling of confidence, that this search 
identified the "best" references, and we h*ve .evidence that it; did not 
identify all references. However, the on-line computer search was the only 
practical means -of providing an access to materials across a broad^range of 
possible topics as itiiyxpensive in terms of actual *cost and necessary 
personnel time. . ‘ * . . 

V -2. On-line computer search of Library of Congress in intensive search 
topics. Generally, the on-line libcon search was time consuming,- scattered, 
and unclear.. It was time consuming with respect to appointments for 
strategy building (collecting and collaborating descriptors), retrieval of 
responses, and‘ordering materials not available through the University of 
Oregon library.. The Libcon search w^is-,scatters, as descriptors were used 
uh i rance^ an PYnaminn nf the oripinkl S"tr at c 2 v ^ i.e. . reading —4 liter*- 


at UlU- w V vu X'"'------- * ' ' 

search resdlts were unclear and needed to be searched manually to assure 
appropriate material selection. The search itself was of little value as 
a total of 29 possible sources yielded only one usable^ selection to be found 
in the University of Oregon library. This would cause an assessment of the 
Libcon search for’ tl\is project's usability to be negligible. 

3. Manual Search of ERIC (RIE an.d CIJE) ^intensive search topic^. 


Because of reservations concerning j£^e comprehensivenessyot ccwiputer searcninj 
'and the limited topic areas which mst-ie a manual search feasible, wd conducted 
a manual ERIC search. The search consumed about 12.hours of time, but it 
enabled a concurrent beginning of the screening process. 

4. Consultation with experts vks both a check of the comprehensiveness > 
of the computer Library of Congress search and the manual ERIC search. We 
found this particularly" valuable for the Competency-Based Education topic 
as it produced not only references but also suggestions for other /approaches 
to information (see CB E implemeita^tions section). ' 


ve sfiarcn topics, 
ybt computer searching 
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Analysis Methodoloc 


larfsi: 


Evaluative comments on the anartysis strategies are limited by the change 
in. topic areas, Tl^e planned and partially developed analysis strategies 
appear more applicable to policy -interpretations of research implications 
than to straigljt^yntheses research. " 

1. Conceptual framework . This approach was used in the preliminary 
analysis and was expected to be used as a screening device for materials 
collected for the annotated bibliography. However, the dearth of materials 
.in the policy interpretations levdls (particularly level 4) led to abandon¬ 
ment of the strategy.* We would suggest that this fact has implications for 

1 _ . * -» , - _i_ _ a. i_-: _ ..l: ~_i 


ment or xne strategy . wuuiu ^ 

the RUX" in .terms of Encouraging^or producing analytic synthesis which clearly 
spell out .alternatives and policy issues which accompany these alternatives. 
Such an approach appears valid even,when priority areas are such as basic 
skills rather than definite policy areas .such^as educational equity. 
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Noty: Although the framework was not used in the general analysis, 
some use was made in screening ana some\referfences are found in^fhe^Valuative 
information section on individual annotated bibliography entridS. We found 
it to be a somewhat’us.e£ul tool for.this purpdge. , *. 

Information on implications check uistl Each annotated,bibliography 
entry has a'format which assesses' the alternatives discussed-in the synthesis , 
and the types of implication on which information is proviaed. And.a number 
'of entries in-this section have not been completed either.because it was*not ~ 
appropriate or because it was not possible to \infer the information from 
r reviewing‘the document. In few instances was this information succinctly 
stated. Again, we feel that this kind ofc information should be clearly 
stated in any synthesis or interpretation of r^s^arch iii order for the synthesis 
to be useful to'educational practitioners or decision makers. We would suggest* 


that RDX encouraged .or sponsored syntehsis should include these kinds of 
information-. * . . \ * * * 


